feat(ai): Add .md extension to provide pages in markdown for LLMs #13994
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![semgrep-code-getsentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
Bundle ReportChanges will increase total bundle size by 30.67kB (0.15%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sentry-docs-client-array-pushAssets Changed:
view changes for bundle: sentry-docs-server-cjsAssets Changed:
Files in
Files in
Files in
App Routes Affected:
|
chargome
left a comment
chargome
left a comment
There was a problem hiding this comment.
Implementation looks pretty solid to me!
The script adds more than 3min to every build now, which will only grow in the future.
@BYK just thinking loud but do you see any chance we could create these at runtime on request? I imagine performance is not the biggest concern for these pages?
And last point we probably don't want these pages to show up in google, can we add the according meta tag for these pages?
next.config.ts
Outdated
| }; | ||
|
|
||
| if ( | ||
| !process.env.NODE_ENV && |
There was a problem hiding this comment.
m: Is this one intended? This condition will always fail, even when NODE_ENV is production
There was a problem hiding this comment.
Intended as otherwise yarn build fails locally, asking for these DSNs. That said I think the negation at the front is incorrect. I wonder how this worked locally.
There was a problem hiding this comment.
Okay reverted this as next build sets NODE_ENV to production by default if it was not set 😮
Added a yarn start:dev command instead to do what I needed.
removing leftover log text
Yeah, we have 8k+ files so it takes a while. That said I do not think the build time will grow significantly as even if you added 100 more docs it's still about 10% of the total docs we have and need to walk through.
As far as I'm aware Next.js does not allow the
Where should I add these tags? These pages are not linked from anywhere (except from the link I just added and I'll add |
Is the .md essential for LLMs? Otherwise we could possibly just add a search param
Nevermind, I thought the HTML came from the server here – ignore my comment 👍 |
chargome
left a comment
chargome
left a comment
There was a problem hiding this comment.
Anyway I don't want to block this, we can look into optimizing this 🚀
Yeah we want the I'll work on a patch to speed things up using NextJS build cache. |
…3994) Adds support for `.md` at the end of every pre-rendered path. Does this by a hack where we rewrite those paths to `public/md-exports/...`. The contents of this directory are generated _after_ `next build` by scraping all html files under `.next/server/app` Not ideal but looks like the easiest path for now. --------- Co-authored-by: Burak Yigit Kaya <[email protected]>
…3994) Adds support for `.md` at the end of every pre-rendered path. Does this by a hack where we rewrite those paths to `public/md-exports/...`. The contents of this directory are generated _after_ `next build` by scraping all html files under `.next/server/app` Not ideal but looks like the easiest path for now. --------- Co-authored-by: Burak Yigit Kaya <[email protected]>
Adds support for
.mdat the end of every pre-rendered path. Does this by a hack where we rewrite those paths topublic/md-exports/.... The contents of this directory are generated afternext buildby scraping all html files under.next/server/appNot ideal but looks like the easiest path for now.
Also added a "View Markdown version" link to pages: